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ABSTRACT 

In the area of primary health care, there is a high demand in Brazil of permanent education and qualification of 
professionals who work in this field. Besides, nowadays it is a consensus that education can be benefited by the use of 
mobile devices, especially due to the possibilities of browsing, use and of easy access to different resources. In this 
context, this article presents an evaluation experience of usability of a mobile system for the access to open educational 
resources in the area of health. The results point to the construction of a simple application and of to easy access to the 
objects available and are of utmost importance to their evolution in the aspect of user experience. 
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1. INTRODUCTION 

In Brazil, the health services are organized in assistance levels: primary (considered the users’ preferential 
gateway in the health care services network), secondary (specialized services) and tertiary (hospital services). 
The primary health care is considered to be a set of actions, in the individual and collective spheres, which 
“comprises the promotion and protection of health, the prevention of dangers, the diagnosis, the treatment, 
the rehabilitation and the maintenance of health. And it is developed through the exercise of management and 
sanitary practices, democratic and participatory, in the form of team work, directed to populations from well 
delimited territories, for which they take sanitary responsibility. It must solve the most frequent and relevant 
problems in the territory” (Brazil, 2011). 

When the technology to support health is applied nationally, it is an efficient way to universalize the 
access to health and education. The use of mobile applications, for instance, makes it possible promoting 
distance health actions and aims to: (i) expand the access to educational actions, both to workers and the 
assisted population, especially by using guidelines available in easy language, with elements of games 
(Marcos et al., 2006); (ii) increase the efficiency of the attendance in the health units; (iii) minimize the 
spontaneous demand in the hospitals; (iv) expand the service coverage; (v) enable the screening of medical 
consultations; (vi) reduce the waiting time between consultations; (vii) and make registrations of the services. 

The use of mobile devices as learning tools in health tends to contribute to the reduction of social 
expenses, because it promotes the educational process in the field, during the professional practice, without 
the need of removing the professional from his work place for training and qualification (Rowe et al., 2007). 
This practice has been increasingly seen as a facilitator to the learning process in distance (Rneebone et al., 
2008) (Marcos et al., 2006), especially in developing countries, due to the decentralization of the 
management of health services, which is common in those countries. 

It can be noticed, therefore, that the use of mobile devices in health plays an important role in supporting 
health services, contributing, additionally, to the formation of human resources in the area, and enables the 
reach of services to countryside areas, by the application of telehealth, especially the tele-education. 
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Besides the educational aspects, the use of mobile technologies benefits and enhances, in the scope of 
professional qualification, the convergence of medias, exploring the diversity of resources in mobile devices. 
Thus, after this introductory section, Section 2 presents correlated works which reinforce the results achieved 
in the suggested solution. The application to support the offline teaching-learning, is presented in Section 3. 
Section 4 shows the process used to evaluate the system under criteria of quality and usability. Section 5 
presents the results. Section 6 makes final considerations and presents ongoing activities. 


2. RELATED WORKS 

Some works deal with proposals of architecture and modeling of systems which meet the requirements 
related to connectivity and access to educational resources, whether via mobile applications or via web. The 
work of Marcos and other authors (Marcos et al., 2006), for instance, suggests architecture of access to 
repository of learning objects through mobile devices. Its motivation lies on the necessity of m-learning 
applications integrated to repositories. However, this architecture has not been fully implemented. 

In the perspective of connectivity, Luz and Fonseca (Luz and Fonseca, 2013) presented a support tool to 
collaborative learning which enables the sending and storage of data between distinct devices even without 
internet. For this purpose, the concept of ad hoc networks is implanted, in which they depend only on the 
existence of devices with Wi-Fi boards which work as routers. 

The work of Ferreira and Castro (Ferreira and Castro Jr., 2013), in turn, presents questions of 
synchronism, aiming to enable possible off-line access to educational resources in virtual learning 
environments. Even though the work does not focus on repositories, its relevance is justified by the 
architecture proposed, based on synchronization of files and database, designed for situations of instability in 
internet connection. 

Another relevant aspect is the matter of recommendation of materials, strongly approached by Reis and 
Barrere (Reis and Barrere, 2014). In this work, it is proposed an architecture which enables a professor to 
provide contents for specific groups, as well as automatic recommendations of content based on 
characteristics of the user and context information. 

It is worth highlighting that the other works bring some interesting applications which emphasize the 
importance of repositories of educational resources as support instruments to professional qualification. The 
proposal of the International Database of Educational Objects (Rodrigues et al., 2012) is one of them and 
brings a repository of educational objects to web environments, whose intent is to cover a unified space of 
access to materials produced by teaching institutions around the world. We should also emphasize, as a 
differential, the importance of processes of validation of contents before they are made available. 

Another work, strongly related to this article, is the Collection of Educational Resources in Health - 
ARES (Brazil, 2015), whose aim is to provide educational resources produced by higher education 
institutions for health professionals. In the context of this article, ARES is used as a subcomponent of the 
system presented in this paper, which will be presented with details in Section 3 of this article. 


3. DESCRIPTION OF THE SYSTEM 

It consists of a system, which aims to provide educational resources to the health professionals. Thus, just as 
in any other service and system of telehealth, there is the need of assuring the authenticity of the available 
content, because such information directly implies attention to the life and health of patients and population, 
besides the other legal, ethic, and professional requirements which are common in telehealth in general 
(Newton, 2014). In addition to these requirements, there is also a demand of monitoring and controlling the 
access of users’ health professionals and allowing that the resources continue available for offline access. 

The objectives mentioned and the other functionalities (detailed in this section) meet two main reasons: 
(i) the requirements of an efficient system to provide educational resources to health professionals and (ii) 
respect the governmental and institutional requirements of the public entities involved (Brazil, 2010) (Brazil, 
2011 ). 
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3.1 Description of the Architecture 

The architecture of the system is based on a client-server model, with the presence of an intermediate server 
(MServer), playing a role of processing information for content on lending (present in the Collection of 
Educational Resources in Health - ARES) as part of the service, in addition to a connection with a federate 
network of servers (National Registry of Health Professionals - CNPS) to carry out the authentication in 
different databases. It is important to remember that the servers CNPS and ARES are independent and they 
are already operating (Brazil, 2015). 

For the user, the system presents itself as a mobile application, in which allows the access to a set of 
educational resources in different basic formats (audio, video, texts, and images) available for download and 
offline visualization (in mobile stage). Thus, the access and utilization of synchronized resources allows the 
user a more homogeneous interaction with the structure, where the content presented tends to be the same, 
with the access being done from different devices. The mobile application was built on Android platform. 

3.2 Description of the Functionalities 

After requirements elicitation process, validation with health professionals invited, evaluation of visibility 
and prototyping, the main requirements (essential) and more urgent were prioritized, composing the initial set 
of functionalities of the point of access and of the system: 

User authentication: The user must access the resources of the system and the contents of the 
application after a federative authentication. 

Provision of educational objects: The system provides the educational objects in formats compatible 
with the mobile devices. The educational objects were, initially, audio, video, text (in PDF), and images. 

Offline access: The system must allow the access to the downloaded resources in sessions without 
internet connection, even though there must be a session time for authentication during the offline use. 

Search for educational objects: It must be possible to search for educational objects, both among those 
which are present online and those already downloaded by the user (in offline sessions). The search may be 
done by themes, formats, institutions, and others, in addition to search keywords. 

Data synchronization: The same user must have access to the downloaded educational objects, even if 
the access is done from different devices. 

Monitoring of access and use: The system must monitor the users’ actions, the operation of the device, 
and the conditions of connection, thus allowing the system of recommendations to be built and improved. 
Aside from making it possible the production of statistics of use, for the purpose of governmental 
management in the context of public education and health. 

Security of transmitted data (non-functional requirement): The transmitted data related to users, 
health professionals, both at the moment of authentication and in other transfers of protocol data, must be 
properly covered and encrypted. 

Authenticity of information (non-functional requirement): Any resource which is accessed through 
the system must be originated from a reliable repository. Thus, it is possible to guarantee that the content 
presented was properly evaluated and authorized by a competent content-producer team. 

3.3 Application’s Interface and Use Cases 

The application was developed and tested, aside from the main functionalities of the intermediate server, thus 
putting the system in a functional stage ready for evaluation. The server for tests of authentication with data 
was left in operation for security reasons. On the “Authentication” screen the user’s authentication is done, 
with the professional’s register data. If there is not a register, the user is directed to a registration page. 
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1. Atengao integral a saude da mulher para enfermeiros 

Tipo: application/pdf Tamanho: 4.32MB 
Data de publicagao: 28/06/2014 

No decorrer do estudo desta unidade, veremos como & importante o papel da equipe, 
em especial da enfermagem, para a observagao dos aspect os fisiologicos normais nos... 



Abrlr 


2. Atengao integral £ saude da mulher para medicos 

Tipo: application/pdf Tamanho: 4 32MB 
Data de publicaqfio: 28/06/2014 

No decorrer do estudo desta unidade, vieremos como e importante o papel da equipe. 
em especial da enfermagem, para a observagao dos aspectos fisiolbgicos normais nos... 


Abrir 


3. Sistemas de Informagdes Geograficas e analise espacial na Saude... 


d O □ 


Abrir 


Figure 1. “My Archive” Screen 

After the authentication, the user starts a session on “My Archive” screen, where he can view the objects 
already accessed which are present in the device. It is possible to filter the objects by format, run a search 
among the downloaded materials, filter by themes, see the details of an educational object (such as 
description, author, date, institution, and file size) and open the resources. 



Figure 2. “UNA-SUS Archive” Screen 

It is possible to browse to the tab “UNA-SUS Archive” (if there is an internet connection). The browsing 
can be done among the objects available for download in the repository. It is likewise possible to filter by 
themes and formats and run a search among the objects in the archive. The details of the objects are presented 
in this screen. 


4. SYSTEM’S EVALUATION PROCESS 

The proposal of this research, which has been developed and evolved, is divided into three big stages: 
conception, development, and evaluation. In order to guarantee the correct execution of the methodology 
proposed, the management is based on the practices of the PMBOK Guide (Project Management Body of 
Knowledge) (PMI, 2013) while the process of development of the solution is based on the agile framework 
Scrum: planning, dynamic and adaptive control based on the execution of the activities proposed. 

The conception stage comprised bibliographic review, understanding of the problem to be tackled, gaps 
identified, and delimitation of the scope. A study of mobile applications applied to health, aiming at defining 
the proper architecture to the solution was carried out. The requirements were evaluated with the support of 
health professionals acting in primary healthcare. At the end of this stage, a concept proof was idealized and 
a prototype of the system was built for the initial and real analysis of the solution proposed. 
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The development stage comprised the selection of the technologies, platforms, and devices for the 
modeling of the requirements raised/defined. The intention was to adopt free platforms and technologies, 
because they favor the dissemination of the use in public services of health, and also facilitate the integration 
with other systems. 

And finally the evaluation stage, which consists of (i) the capture of impression and experience of the 
users; (ii) the elaboration of simulations and empirical studies. The first part will be described in this article. 
The second experimental stage, in progress, is where the system proposed will be evaluated integrated to the 
module of decision, aiming at finding points of improvement and evaluate the impact of the practical 
application to the remote health and community services in real situations. 

With the purpose of capturing the impression and experience of users, a test of usability was planned. 
Usability is a component of the acceptability of a system. Authors like Nielsen and Sommerville 
(Sommerville, 2010) (Nielsen, 1993) affirm that the acceptability is a characteristic of the system, which 
refers, basically, to informing if the system is good enough and meets the specifications of the users. Nielsen 
(Nielsen, 1993) presents the usability as composed of five components: Easiness of Learning, Easiness of 
Memorization, Efficiency in use. Few Errors, and Subjective Satisfaction. 

The tests of usability for mobile applications are done in laboratories or in field. The tests in laboratory 
normally use simulators or the device itself. It is important to register that they are less expensive, but in 
contrast they come up short of the expected, because they do not use the applications in a real environment 
with access via mobile operator, for instance. In order to verify if the components of usability were met, 
measurements were defined, such as: time users take to complete the tasks and number of errors in the 
execution of tasks. 

4.1 Capture of Impression and Users Experience 

For the evaluation of the application and of the continuous actions, the following indicators were defined: (1) 
Learning: evaluation of the participants profile (players involved) in the use of the technological platform, 
based on structured surveys and interviews; (2) Quality of the learning objects: continuous evaluation of 
the insertions of technology from the point of view of the population directly favored; (3) Quality of the 
system: evaluation of the system, in such a way to define points of improvement, especially in aspects related 
to the usability, in other words, if the learning object in the mobile system is easy to be used. 

In the evaluation of the application, we considered: a) Performance: aspects related to the minimum 
requirements of hardware, because of the product lightness; b) Connectivity: it will access repository of 
scenarios and data of the user for the synchronization and also in offline mode; c) Usability: the scenarios 
will be for the use in a self-instructional way; d) Portability: capacity of usage in different mobile devices, 
whether in smartphones or tablets. 

The process of capturing of the user’s impression and experience happened according to the following 
flow of activities: 

1. Survey Application - Intending to identify the profile of the participant, under aspects related to the 
use of technologies and social media, the survey, composed of four questions subdivided in up to five items 
each, was applied. 

2. Orientation to the User - The second step was represented by orientation to the user, in order to 
contextualize the applicability of the system. 

3. Execution of The Task List - After the orientation, the user received a list of eight tasks which should 
be accomplished and in any time questions may be reported. 

4. Evaluation of the Test - In this activity, after the accomplishment of the task list, the user was invited 
to evaluate the way the tests were carried out. Questions related to the easiness of use, disposition and 
organization of information, layout of the screens presented, nomenclature, messages of the system, and 
assimilation of the information tackled were numbered. At the end. the user presented a general impression of 
the test, if it was Interesting or Dull. 

5. Suggestion of Improvement and New Requirements - The last activity had the support of the 
participants, in the sense of suggesting new requirements and improvements to the application. 
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4.2 Elaboration of Simulation and Empirical Studies 

The set of activities related to the evaluation of the application, in real situation, is in the final stage of 
planning. The evaluation will be a systemic and a continuous process which will involve: (i) Continuous 
evaluation of the learning objects produced and of the professionals involved in the execution of the usage 
scenarios; (ii) Self evaluation of by the students and professionals involved; (iii) Evaluation of the mobile 
technology used, emphasis on the impact of its use within the educational and assistance contexts. 

The team of execution will be responsible for the evaluation of the application, encompassing the analysis 
of the technologies, as well as the educational resources available and accessed. The evaluation of the system 
will be done during and after the provision of learning objects, aiming at making adjustments and 
improvements. 


5. RESULTS OF THE EVALUATION 

In this section, the results related to the execution of the activities of capture of impression and experience of 
users will be presented, according to what was presented in the subsection 4.1., items 3 and 4, Execution of 
the Task List and Evaluation of the Test, respectively. 

A set of 7 (seven) professionals, being 2 physicians of family health, 2 professionals in information 
science, and 3 professionals of information technology participated in the test. The professionals were guided 
to execute the following tasks: 

• Task 01 - Initiate the application; 

• Task 02 - Run authentication; 

• Task 03 - Identify if there is any video in “My Archive”; 

• Task 04 - Download from UNA-SUS an audio object; 

• Task 05 - Access additional information of a text object in “My Archive”; 

• Task 06 - Play the downloaded audio object; 

• Task 07 - Find the theme “Children’s Health” in “UNA-SUS Archive”; 

• Task 08 - Filter the theme “Children’s Health” by audio format. 

During the execution the time of execution of each task, the errors made in the execution were observed 
through filming and filling of forms. The average of time spent in the execution of each task is presented in 
Figure 03. Worst time: 204 seconds (task 4), best time: 54 seconds. 



Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 


Tasks 

Figure. 3. Average of time spent in the execution of task 
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Table 1 brings a view of the quantity of errors made by each of the 7 participants in the execution of the 
tasks. 


Table 1. Quantity of errors per execution of tasks by professional 



Tasks 

Users 

1 

2 

3 

4 

5 

6 

7 

8 

User 1 

0 

- 

0 

1 

1 

0 

0 

1 

User 2 

0 

- 

0 

2 

1 

0 

0 

0 

User 3 

0 

- 

0 

1 

0 

0 

0 

0 

User 4 

0 

0 

0 

0 

0 

0 

0 

1 

User 5 

0 

1 

8 

- 

5 

1 

- 

- 

User 6 

0 

0 

0 

3 

1 

0 

0 

0 

User 7 

0 

0 

0 

4 

0 

0 

0 

0 


Task 4 appears to be the most complex, because it presented the largest number of errors made per 
participant. In Figure 3 we can observe that task 4 had the longest time of execution, and it may be a 
consequence of the errors made during the execution. At the end of the accomplishment of the task list, the 
participants were invited to evaluate the system. The results are presented in Figure 4. Best score: 5, Worst 
score: 0. 



0,5 


I I I I . I I 



Figure 4. Evaluation of the System by the Participants 

In general, the participants emphasized, as positive aspects, (i) the simplicity of the application; (ii) clear 
and simple layout; (iii) the alternation between “UNA-SUS Archive” and “My Archive”; and (iv) quick view 
of the themes. Difficulties were observed and reported during the execution of the task list. The most 
recurring were (i) the functionality “return” did not work; (ii) canceling of download of educational 
objects/resources; and (iii) exclusion of files initially downloaded to the device. 

Based on these results, actions of improvement to the application were defined and are were executed. 
Especially questions related to (i) the standardization of the search area of educational objects/resources - 
using Google standard and material design; (iii) the cancelling of the functionality of download; and (iii) the 
sending of messages of loading objects in the application. 
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6. FINAL CONSIDERATIONS AND ONGOING ACTIVITIES 

This paper presented an application developed initially for Android system, whose object is providing 
educational resources and objects, in offline mode, to health professionals. The article brings an account of 
an experience acquired during the execution of tests of usability of the application in laboratory. 

The results obtained were very interesting and contributed greatly to the evaluation of the functionalities 
and usability of the system. Indeed, new ideas were captured for the improvement of the graphic interface. 
The simplicity of the application was considered one of the most relevant aspects, reinforcing the idea that 
browsing must be as simple and intuitive as possible. 

The application was evolved according to the results taken from the evaluation presented in this work. 
Thus, improvements in the interface with the user are were implemented. Besides, the application is going 
through functional adjustments, the most important one being the possibility of taking courses in offline 
mode. 
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